System Testing and Program Verification

The effectiveness of black-box system testing can be increased by automatic program verification techniques. For example, the redundancy of a test case can be detected by static analysis; the analysis must be applied to a program in the ‘whitebox’ layer of the system under test (e.g., in the setting of GUI testing, to the program which defines the event handlers). We will investigate the question of how automatic program verification techniques can be used to reduce the cost of testing and at the same time provide a guarantee for test coverage. Overview. Testing and verification are two sides of the same coin. The corresponding techniques can complement each other. In particular, automatic program verification techniques can be used in system testing for a more informed selection of test cases. We have instantiated the idea for GUI testing, where a system is tested through a sequence of events which are triggered via a graphical user interface (GUI). As described in [APW14, AEFAP14, ABSP12, APB12], we apply a static analysis to the event handler programs of the system. The static analysis infers verifiable statements about the runtime behavior of the programs and thus helps to determine the relevance resp. the redundancy of a test case (where relevance and redundancy have a precise, formal sense). We will next explain the context and the approach in greater detail. The main challenge in GUI testing is to generate test cases selectively. So-called iterative approaches as in Gross et al. [GFZ12] and Mariani et al. [MPRS12] generate test cases on-the-fly (i.e., while executing the system under test). Here, the size of the test suite is a priori unbounded (in practice, timeouts are used for the overall test). In contrast, non-iterative approaches generate test cases offline (before executing the system under test). Here, the size of the test suite can be bounded a priori (e.g., by fixing the length of event sequences). However, many event sequences in the test suite will not be relevant for detecting bugs (e.g., because the events in the sequence do not relate to each other), and many event sequences will not be feasible (e.g., because the button to trigger an event in the sequence is not visible). So-called black-box techniques avoid many infeasible test cases by using a black-box model such as an Event Flow Graph (EFG); the model is constructed by observing the order of events in sample executions of the system [Mem07]. The above-mentioned approach of [APW14, AEFAP14, ABSP12, APB12] integrates automatic verification techniques with black-box techniques. In a first step, a static analysis on the event handler programs is used to compute sequences of events that are interconnected by def-use relationships. The second step uses the black-box model to turn each sequence into one that is feasible according to the black-box model. The approach thus integrates the information of what is relevant with the information of what is feasible. The original approach in [APB12] only considers sequences that contain a pair of two matching def-use events. Even with that restriction, the number of event sequences can